Tolerating Process Imbalance in Hierarchical Collective Operations

This work studies the temporal imbalance that processes experience when operating on a modern multicore cluster, and develops hierarchical algorithms that tolerate this imbalance. A micro-benchmark is used to investigate the nature of process imbalance with perfectly balanced workloads, and understand the nature of interversus intra-node imbalance. These insights are then used to develop imbalance tolerant reduction and broadcast algorithms, that minimize the synchronization delay observed by early arriving processes. Preliminary results show speed-ups reaching 80x for reduce and 14x for broadcast over a binomial tree algorithm in the presence of high, but not unreasonable, imbalance.